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We consider a coding-decoding scheme which can permit reliable data 
communication at rates up to the capacity of a discrete memoryless channel, 
and which offers a reasonable trade off between performance and complexity. 
The new scheme embodies algebraic and sequential coding-decoding stages. 
Data is initially coded by an algebraic (Reed-Solomon) encoder into blocks 
of N symbols, each symbol represented by n binary digits. The N n-bit 
symbols in a block are transmitted separately and independently through 
N parallel subsystems, each consisting of a sequential coder, an inde- 
pendent discrete memoryless channel, and a sequential decoder in tandem. 
Those coded n-bit symbols which would require the most sequential de- 
coding computations are treated as erasures and decoded by a Reed- 
Solomon decoder. We shoiu that the hybrid technique reduces the variability 
of the amount of sequential decoding computation. We also derive asymp- 
totic results for the probabilities of error and buffer overfioio as functions 
of the system complexity. 

I. INTRODUCTION 

It is well known that the use of block coding and maximum-likeli- 
hood decoding permits transmission of information at rates up to the 
capacity of a discrete memoryless channel with an error probability 
which decreases exponentially with the code block length. 1-4 A discrete 
memoryless channel (DMC) may be an adequate model for some 
types of real one-way digital communication channels consisting of a 
transmission medium, transmitting and receiving equipment and modu- 
lation-demodulation scheme. An arbitrary DMC is assumed to have 
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a P-symbol input alphabet and a Q-symbol output alphabet. During 
each channel use, an input symbol is selected and transmitted and 
an output symbol is received. Successive input-to-output transitions 
are random and statistically independent; the probability that the 
output is symbol j {j = 1, 2, . . . , Q), given that the input is symbol 
i (i = 1, 2, . . . , P), is q tj . (Table I contains a list of the symbols 
used throughout this paper) 

Maximum-likelihood decoding, which is known to be optimum, 
involves the cross-correlation of a received block code word with all 
possible transmitted code words. The number of code words, and 
hence the required number of decoding operations, grows exponentially 
with the block length ; this exponential growth in decoding complexity 
makes maximum-likelihood decoding impractical, even for moderate 
block lengths. There has thus been considerable incentive to find 
suitable classes of codes having nonoptimum decoding schemes, for 
which the complexity (reflecting the number of components and the 
number of decoding operations per unit of transmitted information) 
does not increase exponentially with the block length. 

A number of coding-decoding schemes have previously been pro- 
posed. Among the most widely known are: 

(i) Algebraic coding and decoding schemes. 9, 8 

(n) Elias' iterated coding and decoding. 7 

(Hi) Massey's threshold decoding of convolutional codes. 8 

(iv) Gallager's low density parity check codes. 

(t>) Sequential coding and decoding. 10-12 

For some performance-versus-complexity criteria, one or more of 
these schemes may be well suited. However, lower bounds on the 
performance and complexity of these schemes show that none can 
yield an exponentially low error probability for a rate arbitrarily 
close to channel capacity without incurring exponentially growing 
complexity; Ziv, Pinsker, and Forney have proposed some more gen- 
eral coding-decoding schemes for use with discrete memoryless chan- 
nels. 13-10 The common feature of these schemes and of the earlier 
scheme of Elias is that they incorporate two or more separate stages 
of coding and decoding as Fig. 1 illustrates. 7 The "inner stage" is an 
arbitrary block coding-decoding scheme, generally using maximum- 
likelihood decoding, which has just enough complexity to guarantee 
a fairly low probability of decoding error. Then the chain consisting 
of the inner coder, DMC, and inner decoder constitutes another dis- 
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Computational cutoff rate 
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Number of tree branches per serial block 
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Overall information rate in bits per channel use 

Defined by: S = NS - 1 

Probability of decoding error for one serial block 

Upper bound on P u (e) 

Constants, for a given sequential code 

Sequential decoding error exponent 

Probability of error for a super block 
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Overall block length 

Number of sequential decoding computations to decode the jth super 
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Fig. 1 — Two-stage coding-decoding scheme. 
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crete channel with a low probability of error or erasure. Scrambling 
and descrambling may be necessary to make this new channel mem- 
oryless. The "outer" stage or stages embody available coding and 
decoding techniques with long block length, which drive the proba- 
bility of decoding error down to a negligibly small value with a 
relatively small degree of complexity. The overall block length is the 
product of the block lengths of the individual coding stages, and the 
overall information rate is the product of the individual rates. 

The overall block lengths for these schemes are much larger than 
those known to be necessary to achieve a given error probability with 
a given information rate. However, this penalty, which is reflected 
in increased coder complexity, may be compensated for by the more 
favorable tradeoff between performance and decoder complexity. 

These multistage schemes allow transmission at any information 
rate up to channel capacity with error probabilities which decrease 
exponentially with overall block length (or its square root in Ziv's 
scheme) ; the total decoder complexity may be large but it increases 
only algebraically with the overall block length. Notice that if the 
inner stage uses maximum likelihood decoding in order to achieve 
a low error probability for a rate close to channel capacity, its com- 
plexity increases exponentially with its block length. Thus the com- 
plexity of the inner stage may well dominate the total complexity, 
for rates close to capacity. 

We propose yet another two-stage coding-decoding scheme, which 
we call a hybrid scheme and which is described in detail in Section 
II. The inner stage involves sequential coding-decoding, which is 
known to be capable of yielding exponentially small error proba- 
bility for any rate less than the channel capacity. The decoding effort 
required of the inner stage is actually alleviated by the use of the 
outer stage, which involves algebraic coding-decoding. Section III 
contains derivations of upper bounds on error probability, distribu- 
tion of decoding computation, average decoding computation and 
probability of buffer overflow for the hybrid scheme. These bounds 
display the asymptotic performance capabilities of the scheme. The 
bounds are not sufficiently tight to be useful in obtaining detailed 
performance parameters for actual systems, but must be supplemented 
by simulations. Section IV contains some simple calculations, based 
on a previous simulation, for the performance of a hybrid scheme. 
Before describing the new scheme, we briefly review some salient fea- 
tures of algebraic coding and of sequential coding. 
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1.1 Algebraic Coding and Decoding 

Any algebraic code has an underlying algebraic structure, upon 
which the coding and decoding algorithms are based. 5 For a code 
with block length N, each code word consists of N symbols picked 
from a finite field. Thus the symbol alphabet size must be a prime 
or power of a prime. The channel is assumed to either change a symbol 
to a different symbol in the field with some probability p (thus mak- 
ing an error) or change it to a symbol not in the field with some 
probability q (thus making an erasure), or pass the symbol on un- 
changed with probability 1-p-q. 

Algebraic codes may be put in systematic form; K of the N sym- 
bols in a code word are information symbols and the remaining N-K 
are check symbols. The ratio K/N is the dimensionless rate of the 
code. The required coder complexity is generally proportional to N. 

An important property of an algebraic code is its minimum dis- 
tance, d, which is the minimum number of symbols in which any two 
code words differ. Practical decoding algorithms are available for 
certain classes of algebraic codes with specified minimum distance 
properties. These decoding algorithms generally involve a finite num- 
ber of algebraic (finite field) operations, and guarantee the correc- 
tion of up to T errors and S erasures for any T and S such that 

2T + S S d - 1. (1) 

The best known algebraic block codes are the BCH codes, for 
which both the number of decoding operations per block and the 
number of components vary with N approximately as N log N and 
with T approximately as T log A r , as shown by Berlekamp. 6 A special 
case of BCH codes, involving roughly the same order of decoder 
complexity, is the class of Reed-Solomon (RS) Codes. 17 - 18 A RS 
code can be defined with any rate R and block length N, provided 
that the size of the symbol alphabet exceeds N. It can be shown that 
a RS code's minimum distance is the largest possible, given R and N, 
that is 

d = d m „ = (1 - R)N + 1. (2) 

Reed-Solomon codes are useful where the size of the code's symbol 
alphabet can be large. 

1.2 Sequential Coding and Decoding 

Sequential coding and decoding is applicable in principle to any 
DMC. Sequential coding is also known as tree coding. 10-12 Included 
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in the class of tree codes are the easily implemented convolutional 
codes. 10 

A sequential coder accepts a sequence of consecutive binary infor- 
mation digits and, for each, generates v channel input symbols. Cod- 
ing is sequential; each channel input symbol depends only on pre- 
vious binary input digits. 

Implicit in the structure of a sequential coder is a tree, as typified 
in Fig. 2 for v = 3. Each branch is labeled with v channel input 
symbols. A sequence of binary inputs to the coder is conceptually a 
sequence of directions which sequentially steer the coder along a path 
(called the correct path) starting at the origin of the tree. Successive 
branches along the correct path are transmitted over the DMC as 
^-tuples of channel symbols. The rate of the tree code in bits per 
channel use is r = 1/v. If a rate r = u/v is required, bits entering the 
coder would be grouped into tf-tuples, and there would be 2" branches 
stemming from each node of the tree. 
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Fig. 2 — Tree structure of a sequential code. 
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Sequential decoding is a form of probabilistic decoding, which is 
applicable to tree codes. It is termed "probalistic" because the general 
decoding procedure applies to any randomly selected tree code and 
because the decoder is guided to a final decision by probabilistic 
considerations rather than by a fixed sequence of algebraic operations. 
A sequential decoder implicitly contains a copy of the tree, and must 
hypothesize a path through the tree, starting at the origin, which 
with high probability is the correct path. 

The Fano sequential decoding algorithm is a specific sequential 
tree search procedure which is efficient, practical to implement, and 
is amenable to analysis. 11, 12 The decoder examines received branches 
successively, makes tentative hypotheses for the corresponding branches 
of the correct path, and advances along them through the tree, if 
their likelihood, measured by an appropriate "path metric," appears 
high enough. If the current hypothesized path appears not sufficiently 
likely, the decoder retreats one branch and starts searching for a 
more likely path. Thus there is backward and forward searching 
through the tree, with a trend toward the right, as the decoder con- 
tinually extends and revises its estimate of the correct path. If the 
rate r is less than the capacity of the DMC, the Fano algorithm 
sequential decoder can be shown to eventually trace out the correct 
path with high probability. 

The number of branch examinations, or computations done by 
the decoding algorithm to advance one branch deeper into the tree 
is a random variable. Analysis and simulation have shown that its 
mean is bounded, independent of the coder complexity, only if the 
code rate r is less than a "computational cutoff rate," R com v> which is 
characteristic of the channel and is always less than the channel 
capacity. 

Since the rate of transmission and the decoder's operating speed 
are fixed, a buffer must be provided at the decoder to store arriving 
branches which accumulate during periods of intensive tree searching. 
The buffer is necessarily of finite size, and hence may overflow if a 
span of received branches requires an unusually large amount of 
computations. Buffer overflow is catastrophic, since it is accompanied 
by loss of data and subsequent disruption of the decoding process. 
It is generally the most prevalent mode of failure in systems which 
use sequential decoding. 

Restarting the decoding process after an overflow occurs is gener- 
ally possible only if the sequence of transmitted channel symbols is 
divided into blocks which are coded and decoded independently. That 
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is, at regular intervals, the coder starts afresh at the tree origin and 
erases its memory of previous information bits. Then if an overflow 
occurs, decoding can resume at the beginning of the next block. 

It will be shown that the hybrid coding-decoding scheme described 
in the Section II reduces the severe variability in decoding effort that 
is characteristic of sequential decoding, and furthermore, that for 
any rate up to channel capacity, the probability of decoding failure 
(error or overflow) asymptotically decreases nearly exponentially 
with the total system's complexity. 

II. DESCRIPTION OF CODER AND DECODER 

2.1 The Coder 

Figure 3 shows the structure of the hybrid coder. We assume that 
N parallel independent DMC's are available, each of which is used 
for transmission once every t seconds. These N parallel channels 
could be created by time-multiplexing a single DMC which is used 
once every t/N seconds. The input to each DMC is from a separate 
sequential coder. The code rate is r = 1/v bits per channel use. Every 
vt seconds each sequential coder accepts a binary input digit and 
generates v successive channel input symbols which, in accordance 
with the tree structure of the code, depend on present and past coder 
inputs. However, each coder's memory of past input bits is erased 
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Fig. 3 — Hybrid coder structure. 
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at nfr-second intervals. Thus, successive blocks of n inputs are coded 
independently into blocks of nv channel input symbols; such inde- 
pendently coded blocks are called serial blocks, and the correspond- 
ing blocks of n coder input digits are called n-symbols. 

If a coder input digit is to be decodable with a low error probability, 
it must affect a certain minimum number of subsequent channel input 
symbols. However since the coder's memory of previous inputs is 
erased at the beginning of each serial block, the final coder input digits 
in any n-symbol can affect relatively few channel input symbols. The 
error probability is kept low by making the last m {m < n) digits of 
each n-symbol a fixed sequence known to the decoder. 12 Then each a 
'priori unknown coder input digit can affect at least mv channel input 
symbols. The last m coder input digits are redundant; the net informa- 
tion rate of each sequential coder is then (l—m/n/)v bits per channel 
use. In general, n is chosen to be much greater than w, so that the 
decrease in net rate resulting from the periodic "^synchronization" 
is acceptably small. 

The N serial blocks simultaneously coded and transmitted in paral- 
lel over the N DMC's comprise a super block. The corresponding set 
of N n-symbols which enter the coders in parallel is called a parallel 
block. NR of the n-symbols in a parallel block are independent sub- 
blocks each consisting of n-ra information bits followed by m known 
bits. The remaining N(l—R) n-symbols in a parallel block are parity 
check symbols generated from the information n-symbols by an alge- 
braic block coder operating on a field of 2" elements (that is, the coder 
operates on n-symbols rather than individual bits) . Each n-symbol is 
made to enter its respective sequential coder serially, as a sequence 
of binary digits at VT-second intervals. 

A parallel block is thus a member of a block code with block length 
iV and a 2 n -symbol alphabet. The code's dimensionless rate is R, and 
the number of words in the code is 2 nNR . 

The overall information rate of the system is 

R' = R(l — m/ri)/v bits per channel use. (3) 

Since each DMC is used once every r seconds, the overall information 
rate is NR'/t bits per second. A source producing information at this 
rate would determine which of the 2 nNR block code words would be 
generated in each ni>T-second interval. 

For moderate-to-large parallel and serial block lengths (greater 
than, say 50) the most eligible available block code would be a Reed- 
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Solomon code, since the required alphabet size is generally large, and 
RS codes have the largest possible minimum distance for given rate 
and block length. The alphabet size must be a power of two and must 
exceed (N + 1) . This imposes a constraint on n, 



n ^ Iog 2 (N + 1). 



(4) 



Typically, m might be between 10 and 100, n might be 10 or 20 times 
m, and N might be between 10 and 1000. Forney 18 has pointed out that 
if n = n'l (n' and I integers) and 2"' ^ N then a RS code of block 
length TV on a field of 2" elements can be implemented more simply as 
I repetitions of a RS code of block length JV on the subfield of 2"' ele- 
ments. Use of this smaller field for algebraic operations makes for 
simpler implementation of the RS coder and decoder. Figure 4 shows the 
structure of a super block. 

The Reed-Solomon coder may be implemented with a number of 
components proportional to N. Each of the N sequential coders may be 
realized as a convolutional coder, constructed from at most n shift 
register stages. Thus, the overall coder complexity is proportional to nN. 

2.2 The Decoder 

Not surprisingly, a decoder appropriate to the two-stage coding 
scheme just described consists of sequential and algebraic stages, as 
illustrated in Fig. 5. The first stage consists of N parallel sequential 
decoders which simultaneously and independently utilize the Fano 
sequential decoding algorithm to decode serial blocks emerging in 



n 

BITS 



-«-o ;.-> 



I 

NR 
BITS 



m REDUNDANT 
BITS 



(^ 





* BITS *^ 


* 






\\\vSS 


1 

N 
BITS 

I 


1 
1 
1 
1 
1 

1 


i 


1 
r 


i 


i 



(b) 



nv 

•< -CHANNEL- > 
SYMBOLS 



n-SYMBOL 



CHANNEL 
SYMBOLS 



f NO-R-) ! 
REDUNDANT 

f B ' TS * 




.SERIAL 
J BLOCK 



(O 



Fig. 4 — Code structure: (a) block of bits entering RS codes, (b) parallel 
block (output of RS coder), (c) super block (output of sequential coders). 
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parallel from the N DMC's. This stage might be implemented by a 
time-sharing technique, in which a single logic unit is allocated to one 
decoder after another in turn. The second stage is an algebraic de- 
coder for the RS code. 

During the decoding of a super block, all N sequential decoders 
attempt to decode their respective serial blocks into the original input 
n-symbols. In general, some serial blocks require more computations, 
and therefore more computing time, than others. After all but some 
fixed number S (S < N) of the N serial blocks have been sequentially 
decoded, the S sequential decoders still at work are halted, and then 
all sequential decoders are free to start work on the ?i-symbols of the 
following super block. 

Meanwhile the present super block is passed on to the RS decoder 
in the form of a parallel block consisting of N — S sequentially de- 
coded n-symbols and S undecoded n-symbols which are treated as 
erasures. If the RS code's minimum distance is d, and no more than 
T of the sequentially decoded n-symbols contain errors, where 

2T + S = d - 1, (5) 

then the RS decoder is guaranteed to decode the parallel block cor- 
rectly, using a fixed number of decoding computations that varies 
roughly as N log N and as T log N. 6 - 18 In this way, those S serial 
blocks which normally would be sequentially decoded last are es- 
sentially all corrected by the algebraic decoder as soon as the first 
(N — S) serial blocks have been sequentially decoded. Thus the 
algebraic decoder's assistance should tend to curtail the very long 
decoding times which occasional serial blocks may require and should 
thereby reduce the chances for overflow of the sequential decoders' 
buffers. 
From relation (2), governing the minimum distance of an RS code, 

2T + S = (1 - R)N; (6) 

the numbers of correctable errors and erasures are proportional to A r , 
for fixed rate R. 

A hybrid scheme closely related to the one described here was de- 
scribed and analyzed in Ref. 19. In that scheme the sequence of chan- 
nel input symbols is not divided into independently coded serial 
blocks. Instead, once the sequential decoding algorithm advances a 
certain fixed number of branches beyond a given n-symbol, that n- 
symbol is considered irrevocably decoded, and thus is presented to the 
block decoder as a nonerased symbol in a parallel block. As in the 
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Fig. 5 — Hybrid decoder structure. 

scheme described here, n-symbols which would require excessive 
numbers of sequential decoding computations may be decoded by 
the Reed-Solomon decoder. The asymptotic bounds on computation 
statistics are essentially similar for both hybrid schemes. The scheme 
described here appears somewhat more practical to implement. Ref- 
erence 19 also describes a simulation of the earlier scheme in which 
there are ten parallel sequential coding-decoding systems, and the 
block code word rate is either 8/10 or 9/10. The outer stage was in- 
tended to correct erasures only. The tail of the observed distribution 
of sequential decoding computation behaved as predicted by the upper 
bound of Section 3.2; the frequency of very large peaks of computa- 
tion was considerably reduced. 



III. BOUNDS ON PERFORMANCE AND COMPLEXITY 

In deriving bounds on the probability of error, distribution of com- 
putation, average computation, and probability of buffer overflow, 
we assume arbitrarily that the RS decoder corrects T = N8/2 — 1 
errors and S — N8 — 1 erasures per parallel block, where < 8 < %. 
Half the RS code's minimum distance is then used to correct erasures 
and half to correct errors. The value of 8 is then fixed by (6) ; 



S ~ 2 + 2AT- 2 



(7) 
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and 8 is essentially independent of the block length N for large values 
of AT. 

Arbitrarily set m/n = 8. Then the overall rate is 

R' = rR(l - m/n) = r(l - 5)(l - 25 + |=J > r(l - 5)(1 - 25). 

(8) 

It will turn out that the performance of the hybrid scheme depends 
on the distribution of computation and on the error probability for 
the Fano sequential decoding algorithm. Previously known upper 
bounds on these statistics are summarized in Appendix A. The bounds 
are on averages over ensembles of tree codes. Following an argument 
of Shannon, one can show that most tree codes picked at random 
satisfy all the bounds at least to within a small constant factor. 1 For 
example, suppose the ensemble averages of error probability and mean 
computation per decoded bit are upper bounded respectively by X 
and Y. Then at least 9/10 of all possible tree codes have error prob- 
abilities less than 10X, at least 9/10 have mean computations less 
than 10Y, and therefore at least 8/10 satisfy both of these bounds. 

The upper bounds on the error probability 20 and on the distribu- 
tion of computation 23 for rates r exceeding 2?com P are known to apply 
also to the ensemble of convolutional codes, for which the coder's 
complexity is proportional to n. This extension to convolutional codes 
has not been analytically established for the distribution of computa- 
tion for rates below -B r0 mp; 21 ' 22 however, it seems a reasonable con- 
jecture that the degradation in performance due to the implementa- 
tion of a tree code by a convolutional code is small for all rates. 

3.1 Error Probability 

From a result of Yudkin, it is inferred in Appendix A that the prob- 
ability p«(e) that a sequential decoder decodes a serial block incor- 
rectly is bounded by a negative exponential function of m, the number 
of redundant coder input bits in each n-symbol. 20 With m = n8, 

Pu(e) < pi(e) = nA. exp [-n8vE u (r)] (9) 

where A e is a constant and E„(r) is a function of the tree code rate r 
and of the transition probabilities of the DMC. The exponent E u (r) 
is positive for any rate less than the capacity of the DMC. It is 
sketched for a typical DMC in Fig. 6. The probability of error p(e) 
for the hybrid decoder is the probability that iV8/2 or more undetected 
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The asymptotically tight Chernoff bound for the distribution of 
sums of binomially distributed random variables may be applied to 
the right-hand side of (10). 10 



Vie) < exp (-N{T e [8/2, p„(e)] - H{8/2))) 
where 



^ vM < 8/2 (11) 



T.{x, y) = -x In y - (1 - x) in (1 - y) 
H{x) = -x In x - (1 - x) In (1 - a;). 
It can readily be shown that for y < x < V2, 

T e (x, y) - H(x) > 0. 
Thus the bound decreases exponentially with A r . Notice that 

a 



dp u (e) 



T e [8/2, P M) < vM < 8/2. 



(12) 



(13) 



Thus, the exponent in (11) is monotone decreasing in p u (e), provided 
that p u (e) < 8/2; therefore pie) can be further upper bounded by 
substituting p' u (e) for p u (e) in (11) 

p(e) < exp (-N{T,[8/2,p'M - H(8/2)}) P L(e) < 8/2. (14) 

The exponent in (14) will be positive if p' u (e) < 8/2 < |. By virtue of 
(9), this will be true if 



n> toEM tn{2nAJby 



(15) 
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Thus p(e) decreases exponentially with iV if (15) is satisfied. But 

5 > if R < 1 
and 

E u (r) > if r < channel capacity. 

Thus, values of r, 8, and n can be found for which the constraint (15) 
is satisfied, while the overall rate, given by (8), is arbitrarily close 
to the channel capacity; that is, 8 arbitrarily close to zero and r 
arbitrarily close to capacity. 

The overall block length is S e = nN. The serial block length n is 
constrained by (15) and by the constraint on the alphabet size of 
an RS code: 

n ^ log 2 (N + 1). (4) 

Thus for fixed overall rate R', and very large values of N, n behaves 
essentially as log 2 N, or at most as log 2 S e . This implies that for a fixed 
rate less than the channel capacity, the probability of error is bounded by 
a quantity that asymptotically decreases almost exponentially (approxi- 
mately as *S,/log 2 S.) with overall block length S, . Notice also that the 
quantity S, is proportional to the complexity of the hybrid coder, if 
the tree codes are convolutional codes. As mentioned earlier, it seems a 
reasonable assumption that the bounds on error probability and distri- 
bution of computation apply to convolutional codes of any rate. 

The choice of T = N8/2 — 1 was arbitrary but convenient. For 
practical systems where N is less than, say 50, it would undoubtedly 
be more efficient to make m large enough that p' u (e) is negligible and to 
use the RS decoder to correct only erasures, that is, set T = and 
8 = N(l - R)- 

3.2 Distribution oj Computation 

A sequential decoding computation is done every time a tree branch 
is examined and compared to a received branch. Let c be the total 
number of computations to decode a given serial block, that are done 
by a sequential decoder operating alone, without aid or relief from 
an algebraic decoder. Appendix A uses the results of References 19, 
21, 22 and 23 to show that the probability distribution function of 
c is bounded by a function which asymptotically is a pareto distribu- 
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tion. That is, 



pr (c^X) < [n a /a A e /X] 



(16) 



where a' = max {1, a}, A c is a constant, and a is the pareto exponent, 
a function of tree code rate r and of the channel statistics. The pareto 
exponent is positive for all rates less than channel capacity, and is 
greater than unity for all rates less than R eomp , which is less than channel 
capacity. The pareto exponent is sketched as a function of r for a 
typical DMC in Fig. 7. Note that the average of c is finite if and only 
if a is greater than one. It is clear that the smaller a is, the slower is the 
asymptotic decrease in pr (c ^ x), and hence the greater is the varia- 
bility of the random variable c. The bound on pr (c ^ X) will be used to 
upper bound the distribution of the number of computations done by 
the hybrid decoder in decoding a super block. 

For analytical convenience it will be assumed that sequential de- 
coding of any serial block within a super block does not start until: 

(i) The preceding super block has been decoded. 
(ii) The entire serial block has been received and stored in the 
sequential decoder's buffer. 

These conditions ensure that successive super blocks are decoded 
independently, and that during the decoding of any super block there 
is no idle time spent by the sequential decoders waiting for new 
branches to arrive. These assumptions can only delay the operation 
of the sequential decoders in our model, and hence lead to a conserva- 
tive estimate of the buffer overflow probability. 

Decoding of a super block is essentially completed when all but 
S of its N serial blocks have been sequentially decoded. The number 
of decoding operations then done by the RS decoder is bounded by 
a fixed quantity, and will be neglected. Accordingly we define C, the 




Rate r (BITS PER CHANNEL USE) 

Fig. 7 — Pareto exponent a for a typical DMC. 
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number of computation units to decode the super block to be the 
(S + l)th largest of {ci, c 2 , . . . , c„}, where c s is the number of com- 
putations that the ;'th sequential decoder, acting alone, would require 
to decode the jth serial block. Then, no more than C computations 
are done by any one sequential decoder during the decoding of the 
super block. One computation unit represents one or more (up to N) 
sequential decoding computations done simultaneously by the corre- 
sponding number of sequential decoders. 

The number of computation units C exceeds X if (S + 1) or more 
of {ci,Ci, ... ,c N } exceed X. From (16), 

pr (c, ^ X) ^ p x = [n-'-AJX]". (17) 

Then analogous to (14) we have, for S + 1 = N 8, 

pr (C ^ X) g exp {-N[T e (8, p x ) - H(8)]} p x < 5. (18) 

A cruder but simpler bound is obtained by bounding T c (8, p x ) by 
—8 In pa,. Thus for p x < 8 

pr (C> X) Z exp [NH(8)]p N x s (19) 

= [A l /Xf Sa 
where 

Ax = exp [H(5)/a8]n a ' /a A e . 

From the definitions of A t , and H(8), and expression (17) for p„ it is 
easy to show that the condition p, < 8 is certainly true if X > A x . 
Also, pr (C ^ X), being a probability, is certainly bounded by unity. 
Thus 

pr (C^X)^ \ [A ^ /X]Ni X > Al . (20) 

I 1 X ^ A, 

Notice that the right-hand side of (20) asymptotically has the 
form of a pareto probability distribution, but that the effective pareto 
exponent is N8 times the pareto exponent for pure sequential de- 
coding. Now, 

5 > if < R < 1 

and 

a > if r < channel capacity. 
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As the overall rate approaches channel capacity, a and 5 both ap- 
proach zero and the "break point" A t grows very large [Ai also in- 
creases as the 1/ath power of log 2 (N + 1) for very large values of N]. 
However, for arbitrarily small but fixed values of a and 8, the RS code's 
block length N may be chosen sufficiently large that the effective pareto 
exponent N8a can be arbitrarily large and hence pr (C ^ X) arbitrarily 
small, for any X greater than A Y . 

For a fixed value of N, the upper bound (20) is interesting only for 
X » A l or for values of a and 5 large enough that N8a » 1. For values 
of X for which (20) is not tight, the probability pr (C ^ X) is upper 
bounded by the probability that the largest of {ci , c 2 , • • • , Cat} exceeds 
X; that is, it is bounded by N pr (C ^ X) where pr (C ^ X) is bounded 
in (16). 

3.3 Average Computation 

Presumably, the average number of computation units done per 
super block is bounded if N8a > 1, even if < a £ 1. This is true, as 
will now be shown. The average of C is written 

<C)uv= EXpr(C = x ) 

-Y-l 

= Z *[P r (C ^X) -pr(C ^ X + 1)] (21) 

-Y=l 

= Zpr(C^Z). 
x-\ 

Then by (20) 

(C) av ^ A, + E (A x /X) Nia . 

X-A,+l 

The sum can be bounded by an integral from Ai to infinity, since the 
integrand is positive and monotone decreasing. 

<C%v ^A l +T [AJXp m dX 

J A, 

N8aA, 



N8a - 1 



< oo if N 8a > 1 (22) 



= j^Z i [exp [H(8)/ a 8]n" ,/a A c ]. 
Thus the average number of computation units per super block is 
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bounded if the effective pareto exponent N8a exceeds unity for any 
overall rate that is arbitrarily close to capacity, if N is chosen sufficiently 
large. 

The bound on (C) BT varies with n as n a ' /a . Note that the number of 
computation units C is a bound on the number of computations done by 
each of the N sequential decoders, and that the number of information 
bits decoded by each sequential decoder per super block is no more than 
n. Thus the average number of sequential decoding computations per 
information bit is bounded by 



(C\ v /n g A 2 n a ' /a - 1 N8a > 1 (23) 



where 



A * = N fol i texP [H(S)/aS]A c ]. 

Since the block code is Reed-Solomon, n is constrained by 
n ^ log 2 (N + 1). The overall block length (reflecting the complexity 
of the hybrid coder) is nN. Thus the minimum possible value of n 
behaves as the logarithm of the overall block length, and the average 
computation per bit increases as the (a' /a — l)th power of the loga- 
rithm of overall block length. Furthermore, if r < iT! eom p then a = a > 1, 
and the average computation per bit is independent of the overall block 
length. 

For rates above i? CO m P , the exponent a' /a increases rapidly with rate, 
approaching infinity at channel capacity. Thus the bound on the 
average computation, although finite, increases very rapidly with rate 
above R comp . The average computation observed in the simulation 
reported in Reference 19 did indeed increase very rapidly with rate 
above R comi> . 

3.4 Probability of Buffer Overflow 

A new super block arrives to be decoded once every nvr seconds. 
Each of the iV sequential decoders is provided with a buffer which 
is assumed to store the latest Bv received output symbols from its 
respective DMC. Since we have assumed that all symbols comprising 
a super block must have been received before any decoding of the 
super block can start, the total storage must be large enough to con- 
tain one or more super blocks, that is, B must exceed n. Whole or 
partial super blocks stored but not yet decoded form a queue. 

If the queue exceeds B/n super blocks (Bv channel output sym- 
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bols per DMC) buffer overflow occurs. We wish to upper bound 
P L (B), the probability that the buffer overflows before the first L 
consecutive super blocks are decoded, given that the decoder starts 
with initially empty buffers. 

Let qi be the number of undecoded super blocks in the queue just 
after i consecutive super blocks have been decoded. Let X t be the 
number of new super blocks which arrive to join the queue during 
the decoding of the ith super block. Because of our convention that 
decoding of any super block does not begin until the entire block 
has joined the queue, the number X { does not include the ith super 
block itself or later super blocks. The random variables X< and q t 
are not necessarily integers, since a fraction 1/n of a super block 
arrives to be decoded every vt seconds. 

When decoding of the first super block starts, the queue consists 
of only the first super block. Just after the first super block is decoded, 
the queue is thus diminished by one but has been increased by X\. 
Thus 

ffl = 1 - 1 + X 1 « X, . (24) 

Just after the second super block is decoded, 

fffi — 1 + X 2 if q x ^ 1 ^25) 



?2 = 

X 2 if g, < 1. 

This is upper bounded by q x + X 2 for any q, ^ 0. Therefore 

q 2 ^ X, + X 2 . (26) 

Similarly, 

\q 2 - 1 + X 3 if q 2 ^ 1 



93 = 

X 3 if q 2 < 1 

^ X, + X 2 + X 3 for any q 2 ^ 0. (27) 

By induction then, 



q< 



^T,X<. (28) 



This upper bound increases monotonically with i. It is clearly a crude 
approximation for large i. However it will turn out to yield a theoret- 
ically interesting upper bound on p L (B) , at least for values of L 
which are small relative to B. 
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p L (B) = pr [(g x + 1 ^ B/n) or (q 2 + 1 ^ B/n) or • • • 
(<Z* + 1 £ £/n)] 
= pr [max \q x , q 2 , ■ • • , q L ] ^ (B - n)/n] 

^pr[E*<^ (B-n)/nj- (29) 

This inequality follows from (28) and the fact that all X ( ^ 0. 

Suppose each sequential decoder is capable of doing up to n computa- 
tions in each ur-second interval, during which time a new branch arrives 
in each buffer. The parameter n must be several times greater than the 
average number of computation units that the hybrid decoder does per 
information bit, if the decoder is to keep up with the incoming data. 
The hybrid decoder is "busy" (doing exactly /x computation units every 
ur-second interval) until it is about to start decoding a super block 
which has not yet completely entered the buffer. From that instant it 
is idle until the entire super block has entered the buffer, at which time 
it becomes busy again. Thus, a busy interval can only be initiated just 
after the arrival of some super block, and can end only upon completion 
of the decoding of some subsequent super block. Suppose that during 
a particular busy interval, the vth through {v + 77) th super blocks are 
decoded (v, rj integers; L ^ v ^ 1, ij ^ 0). Let C, be the number of 
computation units to decode the Zth super block. Thus ]^#-J ^< k * ne 
total number of computation units done during the busy period. The 
first new super block to arrive during the busy interval arrives after rj 
computation units have been done; thereafter, super blocks arrive every 
tin computation units. Thus (l/im) ^L' t V, C t super blocks arrive during 
the entire busy interval. Successive busy intervals do not overlap, and 
therefore until the Lth super block is decoded, 

Z X t g (l/jin) £ C, . (30) 

i-i t-\ 

Thus, from (29) , 

p L (B) £ pr [ g C, ^ n(B - n)] • (31) 

Since coding and decoding is independent from one super block to the 
next, the random variables \C , , I = 1, 2, ••■ , L\ are statistically 
independent, and have a common cumulative probability distribution 
function which is bounded by the asymptotically-pareto distribution 
function (20). 
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The probability that overflow occurs before the first block is de- 
coded is 



Pl (B) ^ pr [C, h (B - n)] < \-^= 



n). 



(32) 



In appendix B an upper bound is obtained for the probability distri- 
bution of a sum of L statistically independent pareto-distributed ran- 
dom variables.* If the distribution of each random variable is upper 
bounded by pr (C, £ X) ^ (A/X)', s > 1 then it is shown that 

pr[ZC ^ y < DL(Ac/yy, (33) 

where D = 1 + e 8 . This bound is valid for values of L which are small 
relative to y ; specifically, for 

{LA/y)(n{y/A'L)(n{ij/A) < e~\ (33a) 

Applying inequality (33) to (31), we obtain the following bound 
for the probability of buffer overflow before L super blocks are 
decoded: 

p<w < M b^J"- NSa > l (34) 

provided that 

»(B-n) { A* ia L J I A, J 

Condition (34a) will be satisfied for values of L which are small 
relative to the product of decoder speed and available buffer size 
fi(B-n). Inequality (34) then indicates that Vl(B) tends to increase 
linearly toward one with L and to decrease asymptotically as the 
negative {NSa) th power of jx.{B-n) . 

The techniques used to bound p L (B) were too crude to yield a useful 
result for small values of p(B-n) or relatively large values of L; if 
condition (34a) is not satisfied, p L (B) can only be estimated by heu- 
ristic reasoning. The waiting line of undecoded super blocks can 
increase during the decoding of a given super block only if C, the 
number of computation units to decode the super block exceeds the 
number of computation units the decoder can do in nvr seconds, that is, 

* Jelinek has given a more easily derived upper bound, which in its dependence 
on L, is at least as tight as our bound for 1 ^ s ^ 2. 80 
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if C > n/x. The probability that the queue increases is bounded by 
(20) with X = rip.. 

pr (C > nfi < {AJmx) Nia . 

If the decoder's speed factor p. is made large enough so that rip > 
(C) av where (C) 11V is bounded by (22), then the probability that the 
queue increases during the decoding of any super block approaches 
zero as A r approaches infinity. Then the queue would be expected 
to remain close to zero most of the time, and consequently the proba- 
bility that the buffer overflows during the decoding of a given super 
block would be approximated by pi(B'), the probability that the first 
super block causes buffer overflow. For this reason, we use pi(B), 
bounded by (32) as a measure of buffer overflow probability. 

It was shown in Section (3.3) that if N8a > 1, the mean computa- 
tion per super block is bounded by 

(0.^^.4,. (22) 

A hybrid decoder which can perform at least (C) nv computation units 
in a ni»T-second interval can, on the average, keep up with the in- 
coming stream of super blocks arriving at ni^-second intervals. A 
necessary condition for np > (C) nv is 

u = Mo Al = Mon «'/«- 1 ^ c exp [H(8)/a8], (35) 

n 

where p.„ is any number greater than N8a/{N8a — 1) . 

Under condition (35) pi(B), given by (32), is bounded by 



Pi(B) < 



_M.(£ - n) 



"jiVJa 



(36) 



A fairly realistic measure of the cost of the hybrid scheme is the 
total amount of buffer storage utilized. If each of the N individual 
sequential decoders has a buffer capable of storing Bv channel output 
symbols, the total number of symbols which can be stored is 

S t = BNv. (37) 

Suppose we set 

B m n(l + e//io). (38) 

Then 



= exp 
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p,(5) < exp [-N8a] 

= exp [-S, da/Bv] 

Ua + !/!)]■ (39) 

For very large values of N (and therefore of S t ), the necessary- 
value of serial block length n increases no faster than log 2 S* to fulfill 
the constraints (4) , (37) and (38) . Consequently, the buffer overflow 
probability pi{B) is bounded by a quantity that asymptotically 
decreases almost exponentially with the total decoder storage S t (that 
is, as S t /log 2 S t ). Furthermore, the exponent in (39) is positive pro- 
vided that S > and « > 0. These conditions may be met for any 
overall rate R' which is less than channel capacity if the tree code 
rate r is less than channel capacity, and 8 is small enough so that 
condition (8) is fulfilled. The derivation of this result suggested that 
best use would be made of a large but fixed amount of buffer storage 
if the number of parallel sequential coding-decoding systems is as 
large as possible, while the amount of storage allocated to each is a 
relatively small fixed multiple of the serial block length n. 

IV. A NUMERICAL EXAMPLE 

The upper bounds of the previous section are generally useful only 
if one is interested in asymptotic performance. Calculation of per- 
formance parameters for an implementable system should be based 
on the results of simulations. In this section we illustrate the estima- 
tion of performance parameters, based on a simulation of a sequen- 
tial decoder. 

Reference 25 describes the computer simulation of a Fano algorithm 
sequential decoder which decodes convolutionally coded binary anti- 
podal signals received from a quantized phase-coherent white gaus- 
sian noise channel. For a convolutional code rate r = 1/7 bits per 
channel use, a signal-to-noise ratio of —6.5 dB, and an 8-level channel 
output quantization scheme, the pareto exponent a was very close to 
unity, that is, R^p was close to 1/7. Other parameters are: 

(i) serial block length n = 360 branches 

(ii) number of redundant branches per serial block m = 24 
(Hi) convolutional code constraint length = 24 branches. 

The net information rate of this system was then 

I 36 ° ~ 24 = 0.133 bits per channel use. 
7 360 
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Assume the following RS code parameters 

(i) Block length N = 31 = 2 5 - 1. 

(ii) Alphabet size = 32 = 2 5 , so that each super block is a se- 
quence of 72 RS code words. 

(Hi) Rate R = 26/31, so that 5 serial blocks out of 31 are check 
symbols. 

(iv) The RS decoder is designed to correct no errors and up to 5 
erasures per RS code word. 

The RS decoder would be easy to implement. A 155-bit register is 
required to store a RS code word consisting of 31 32-ary symbols. In 
addition, circuitry must be provided to solve 5 parity check equations 
to find the values of up to 5 erased 32-ary symbols. Forney has de- 
scribed efficient techniques for finding values of erasures. 16 The number 
of RS decoding operations is on the order of the square of the number 
of erasures which can be corrected. 

Reference 25 shows empirical probability distribution functions for 
the total number of computations per serial block as observed in the 
simulation. For example, for the —6.5 dB channel, the probability that 
c, the number of computations per serial block exceeds 36,000 is approxi- 
mately 10" 2 . Thus the probability pr (C ^ 36,000) that the number of 
computation units to decode a super block exceeds 36,000 equals the 
probability that 6 or more of the 31 serial blocks require more than 
36,000 computations. This probability is obtained from tables (S. 
Weintraub, Tables of the Cumulative Binomial Probability Distribution 
-for Small Values of p, London: Collier-Macmillan, 1963). 

pr (C ^ 36,000) = £ [ 3 ?]p'(1 ~ 2>) 36 ~* = 6 X 10" 7 (p = 10' 2 ). 

Now assume that each sequential decoder is fast enough to do /x = 
50 computations between received branches. Then, up to 360/x = 18,000 
computations can be done by each decoder in the time taken for one 
new serial block to enter the buffer of each; hence if each sequential 
decoder has a buffer with a storage capacity of three serial blocks, 
the buffer storage will overflow (starting from the initially empty 
state and assuming that decoding of a block starts after it is within 
the buffer) if the first super block requires more than 2 X 18,000 = 
36,000 computation units. Then, assuming overflows are rare enough 
to be nearly statistically independent, the buffer overflow probability 
per super block would be about 6 X 10 -7 . Each decoder's buffer stores 
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3 X 360 = 1080 received branches, and the total number of branches 
stored is thus 1080 N = 33,480. The total number of bits (one per 
branch) per super block is 360 X 31 = 11,160. 

Taking nb = m = 24, v = 7, E u (r) log 2 e « R oomp « 1/7, and assuming 
that A, ph 1 and that the upper bound (9) holds for convolutional codes, 
we have a rough upper bound for p„(e), the probability of undetected 
error per serial block. 

p„(e) < 360 X 2~ 24 = 2.23 X HT 5 . 

(In the simulation, none of 1331 decoded blocks contained undetected 
errors.) 

The probability of an undetected error for a super block is the 
probability that one or more of the 31 serial blocks has undetected 
errors; this probability is upper-bounded by 31 X 2.23 X 10~ 5 = 
6.9 X 10 -4 . This probability may be considered too high. It may be 
decreased about 3 orders of magnitude by increasing the value of 
m from 24 to 34. The resulting increase in the serial block length 
from 360 to 370 should cause negligible effect on the distribution of 
computation per serial block. 

The net information rate of this system is rR{n-m)/n = 0.109 bits 
per channel use. It can be shown that the required signal-to-noise 
ratio per information bit is about 4.7 dB above Shannon's theoretical 
minimum for the infinite bandwidth white gaussian noise channel. 

By such simple calculations based on extensive simulations, one 
can optimize the parameters of a hybrid scheme to meet given cost 
and performance criteria. 

V. CONCLUSIONS 

In the hybrid decoding scheme the number of decoding computation 
units per super block is a random variable, reflecting the probabilistic 
character of the sequential decoders' operations. However the pareto 
exponent is proportional to N; the frequency of large peaks of compu- 
tational effort is reduced by algebraic decoding of the occasional serial 
blocks which otherwise would require excessive sequential decoding 
computation. 

It was shown that for any overall information rate that is strictly 
less than the channel capacity, a finite minimum value of parallel block 
length N can be specified such that the average number of sequential 
decoding computations per bit is bounded by a quantity varying as 
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n a ' /a ~ l , where a is the original pareto exponent for the sequential 
decoding components and a' = max [a, 1}. The number of algebraic 
decoding computations per bit is a fixed number which is almost inde- 
pendent of parallel or serial block length. 

It was also shown that for a proper choice of parameters, the error 
probability decreases nearly exponentially with the overall block length, 
and (heuristically) that the probability of buffer overflow asymptotically 
decreases almost exponentially with the total amount of storage at the 
decoder. These results can hold for any overall information rate which is 
strictly less than the channel capacity. 

A rigorous upper bound was also obtained on p L (B), the probability 
that the buffer overflows before L super blocks are decoded. The bound 
is valid for p(B - n) » L, and behaves as L[A l e/pB] Nta for B » n and 
fixed effective pareto exponent NSa. 

The hybrid scheme shares the multistage feature with the schemes 
of Ziv, Pinsker, and Forney. 13 - 1 " In Ziv's scheme, there is an in- 
termediate stage in which errors made by the inner block coding 
stage are detected and treated as erasures. After a scrambling-de- 
scrambling procedure these erasures are corrected by an outer block 
coding-decoding stage. Forney's scheme has two stages; a large alpha- 
bet RS code outer stage corrects errors and/or erasures made by an 
arbitrary inner block coding-decoding stage. Pinsker's scheme utilizes 
sequential coding-decoding for the outer stage. The principle is that if 
the inner stage has a sufficiently low error probability, the rate Rc 0mp 
seen by the outer stage is little different from channel capacity. (This 
is, in a sense, the inverse of our hybrid scheme.) 

In the hybrid scheme described in this paper, the inner and outer 
stages embody sequential (probabilistic) coding-decoding and alge- 
braic coding-decoding respectively. Sequential coding and decoding is 
practical to implement and is efficient for any given DMC, which 
might be created from a physical communication channel by effi- 
cient modulation, demodulation, and quantization. 20 - -' The number of 
computation units per super block is a random variable, reflecting 
the probabilistic nature of sequential decoding and of short-term 
channel behavior. However, the variability of the sequential decoding 
computational load is eased substantially by the outer (algebraic) 
stage. Thus, in contrast with previous multistage schemes, the outer 
decoding stage assists the inner decoding stage, as well as correcting 
its errors. 

Modifications and generalizations of the hybrid scheme are pos- 
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sible. A related scheme, in which channel symbols are not organized 
into independently coded blocks, was studied in Reference 19. An- 
other modified hybrid scheme, falling into the general class of con- 
catenated schemes considered by Forney, is implemented by imposing 
an upper limit X on the number of computations any sequential 
decoder can do on a serial block. 16 Assuming the speed factor fx is large 
enough that X computation units may be done in the time taken 
to receive one super block, no queue of undecoded super blocks can 
build up, and the buffer overflow problem is eliminated. Instead, any 
super blocks requiring more than X computation units are passed on 
to the user as erasures. The probability of erasure is then bounded 
by the right-hand side of (19) with X = X > A lf that is, it de- 
creases exponentially with parallel block length N. 

The multistage approach embodied in the hybrid scheme would 
also appear to be useful for real channels with memory, where errors 
or severe channel disturbances occur in bursts, usually separated by 
fairly long intervals with only scattered random errors. If the N 
serial blocks comprising a super block are transmitted consecutively, 
a burst occurring during the transmission of one or more consecutive 
serial blocks would likely render them nearly undecodable by se- 
quential decoding. Then if the burst did not extend over more than S 
serial blocks, an outer Reed-Solomon or other burst-correcting stage 
could correct the resulting erasures. The application of hybrid or 
other multistage coding schemes to real channels with memory is an 
interesting area for future investigation. 

Any "hybrid" or "concatenated" coding-decoding scheme, incor- 
porating a number of separate parallel coders and decoders would 
likely be orders of magnitude more complex than present day coding- 
decoding schemes for discrete memoryless channels. However the ad- 
ditional complexity may be a tolerable price to pay for the benefits 
of increased reliability and more efficient utilization of the commu- 
nication channel. It is also well to remember that highly complex 
digital systems are becoming increasingly feasible as a result of rapid 
progress in integrated circuit technology. 
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APPENDIX A 

Bounds on Performance for Sequential Decoding 

Various upper bounds on the probability of error and the distribu- 
tion of computation for the Fano sequential decoding algorithm have 
been given in References 20, 21, 22, and 23. All these bounds were 
obtained by random coding arguments, that is, by averaging over an 
ensemble of tree codes with a given probability distribution. The 
results apply to an arbitrary DMC with a P-symbol input alphabet 
and Q-symbol output alphabet, and a transition probability matrix 
{Qij}- We shall summarize some of these previous bounds and then 
shall relate them to the performance of the hybrid scheme. 

Using Gallager's notation, we define the function 

j-i L t-i J 

where 

{Pi} i = 1, 2, . . . , P is the probability distribution on the channel 
input symbols which maximizes E (p). 3 

It can be shown that E (p) is a nondecreasing function of p, that 

E o (0) = 
and that 

lim- E b (p) = C 

P-0 P 

where C is the capacity of the DMC in bits per channel use. 

Any transmitted serial block is a sequence of nv channel input 
symbols which label the corresponding correct path through the 
code tree. A path which diverges from the correct path is termed an 
incorrect path. A sequential decoder makes an undetected error at 
some node lying on the correct path, if the pattern of channel symbol 
transitions causes the decoder to reach the end of the serial block 
while on some incorrect path stemming from that node. One or more 
branches following the node will then have been decoded incorrectly. 
Of the n coder input bits which generate a serial block, m (the final 
m) are known to the sequential decoder. Hence a necessary condition 
for an undetected error to occur in decoding any serial block is that 
an incorrect path exists whose corresponding sequence of coder input 
digits matches that of the correct path in m or more places, and 
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which the decoding algorithm can follow past those m places. The 
probability p ft (e) that this necessary condition is fulfilled for say 
the hth node lying on the correct path has been upper-bounded by 
Yudkin, by random coding arguments. 20 

p k (e) < A. exp [-mvE u (r)] (40) 

where A. is a constant and E u (r) > for ^ r < C„ . The exponent 
EJx) is sketched for a typical DMC in Fig. 6. It is considerably greater 
than the unexpurgated error exponent for block codes with the same 
rate. 3 In fact, E u (r) = E (l) for rates below r = E (1) log 2 e bits per 
channel use. This result for convolutional codes was also shown by 
Viterbi. 20 The probability of error p u (e) for a serial block is upper 
bounded by the probability that the necessary condition for undetected 
error occurs for one or more of the n nodes on the correct path. By 
the union bound, 

PM ^ Zp*(«) ^ nA e exp [-mvE u (r)]. (41) 

Inequality (9) follows from this result with n6 substituted for m. 
Consider tree codes of rate r bits per channel input, where the tree 
extends infinitely to the right. The incorrect subset of the hth node 
lying on the correct path is defined to consist of that node plus the 
infinite set of nodes lying on incorrect paths which stem from the hth. 
node. Let y h be the total number of computations (examinations of 
branches) ever done on nodes within this incorrect ubset. Then y h iss 
a random variable over the ensemble of tree codes and channel transi- 
tion sequences. The sth (s > 0) moment of y h is bounded by a fixed 
quantity A' c for rates r such that 

r < (E (s)/s) log 2 e. (42) 

The quantity A' e is a function of s, r, { p, } and [q ti } . This was established 
for integral values of s by Savage, for all s ^ 1 by Yudkin, and for 
< * ^ 1 by Falconer. 19,21_23 In particular, note that the mean of 7* 
is only bounded for r < E (l) log 2 e. The quantity E (l) for a DMC is 
also denoted by R comtl , that rate below which the mean computation is 
finite. This bound on (y' h ) av leads to an upper bound on the probability 
distribution pr (y h ^ x) by use of the Chebyshev inequality. 21 

pr (y h ^ x) ^ <7;>. v af ' s > ^ 

^ A' e x~' r < (E (s)/s) log 2 e. 
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The pareto exponent a is defined parametrieally by 

r = (E B (a)/a) log, e. (44) 

Then for any c > 

pr( 7A ^ x) ^ (A c /x)' a - e) . (45) 

The right-hand side of (45) is proportional to a pareto probability 
distribution. The positive quantity e may be made arbitrarily small 
by setting A c large enough. Henceforth, we shall ignore e as trivial 
since it would not affect our asymptotic results. Thus, we write 

<Y a *)av < A- (46) 

and 

pr (y k ^ x) < {AJx) a (47) 

where 

E (a)/a = r/loga e, 

where the rate r is in bits per second. The pareto exponent for a 
typical DMC is shown in Fig. 7. The exponent on the right-hand 
side of (47) agrees asymptotically with that of a lower bound on the 
distribution of sequential decoding computation derived by Jacobs 
and Berlekamp. 24 

Let us now relate this upper bound on the distribution of computa- 
tion for the Fano sequential decoding algorithm to the sequential 
decoding of serial blocks in the hybrid system. Only the portion of 
the code tree to a depth n branches from the origin is used to code 
and decode a serial block. Furthermore the last m information digits 
are known to the sequential decoder. Truncating a tree at a depth of 
n branches and making known the final m information letters can 
only reduce the number of branches a sequential decoder must exam- 
ine before completing all computations in the first n incorrect subsets 
of an infinitely deep tree. Furthermore, it can be shown that for 
the Fano sequential decoding algorithm, allowing the decoder to 
search branches beyond depth n cannot reduce the number of com- 
putations ultimately done within a depth of n branches. Therefore, 
if c is the total number of computations to decode a serial block, 
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and 



(e,) "-([S 7j ]l (a>0) ' 



(49) 



The right-hand side of (49) may be bounded with well-known in- 
equalities. 28 



([Ml- 



Z<tX 0<«gl. (50) 

h-l 

{£ itob-r"}' «ii. (si) 



Since 



we have 



or 



where 



(yD« < A:; <a< co, for all A, 

. „ ^ fn4 e a < a ^ 1 (52) 

(C )av ^ -^ 

[n a A a e a ^ 1 (53) 

(c a )„ < n a 'A a e (54) 



a' = max (1, a). 
Then pr (c ^ z) is bounded using Chebyshev's inequality 

pr (c^x) ^ <c a > av af a ^ n m \AJx) m (55) 

where a is given parametrically by r = [E (a)/a] log 9 e. 

APPENDIX B 

Probability Distribution of a Sum of Independent Pareto-Distributed 
Random Variables 



It is required to upper-bound 



where the {C<} are a set of independent positive integer- valued ran- 
dom variables whose distribution is asymptotically bounded by a 
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pareto distribution function 

pr [C. S x] g \ WX) '< X * A (56) 

I 1, < x < A 

where s is greater than one. The following assumption will be found 
necessary 

(A/y) In (y'/A'L) In (y/A) < ±- (57) 

This assumption is tantamount to requiring that y be large relative 
toL. 

We shall split the required probability into two parts, one of 
which is bounded by a union argument, and the other by use of a 
Chernoff technique (Reference 12, p. 97) . That is, we write 



where 



pr I £ C, ^ y\ = Vi + V* 
Vi = P r 2 Ci ^ y; one or more of [d] ^ y\ 

p 2 = pr [ec ^ y> a11 K?«l <y\ 



(58) 



But 



p, < pr [one or more of { C, } ^ y] 

^ Z pr (C, ^ y) (59) 

t-i 

by the union probability bound. So, substituting (56) into (59) , we get 

Pl < L(A/y)' y>A>l ^ Q) 

s > 1. 

The probability p 2 rnay be bounded using the Chernoff technique, 
since each random variable d, being upper-bounded by y, has a finite 
moment generating function. To bound p 2 we first define 

/„ = pr (d = Zi) z t = 1, 2, • • • , y; ^ 

i _ 1,2, .-■ ,L 
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and 

<Kx) =[ 1 X -°- (62) 

to x < 

Then by definition, 



Pa = E/, E/.. ••• £/..* E*«-v|- 



(63) 



We upper-bound the step function *(x) by the exponential function 
exp(Az), where A is an arbitrary positive quantity. We shall later 
choose a convenient value for A. The right-hand side of (63) can 
now be bounded by a product of sums. 

v-i 1/-1 1/-1 r / i \ 

p 2 ^ E /.. E /.. ••• E /.* exp x( E*. - y) 

= exp (-\y) I! | E /., exp (X*,) (64) 

= exp (-\y) [£ /- exp (te)J , (65) 

since the random variables {zi} are identically-distributed. 
Now let 



4, = E/,exp(X2). (66) 

z-l 

This may be expressed in terms of the distribution function pr (C ^ z). 

/, = pr (C = z) = pr (C ^ z) - pr (C ^ 2 + 1) (67) 

So, 

* = E exp (Xz)[pr (C ^z) - pr (C ^ 2 + 1)] 

2 = 1 

= 1 + E [exp (Xz) - exp (X(« - 1))] pr (C ^ 2) 
1-1 

- exp(X(y- l))pr(C ^ y), (68) 



since 



pr (C ^ 1) = 1. 
Taking out the common factor [1 - exp(-A)] and upper-bounding 
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it by A, we get 

i> < 1 + [1 - exp (-X)] 2 exp (Xz) pr (C ^ z) 
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^ 1 + X E exp (Xz) pr (C ^ 2). (69) 

2 = 1 

The function $ is further bounded by employing the upper bound 
(56) for pr (C ^ z). 



^<l + \Eexp (Xz) + X 2 exp (Xz)(4/z)\ 



(70) 



We now express the exponential functions as convergent power series 
and interchange the order of summation to yield 

* < 1 + t X exp (Xz) + £ ^ Z z"- 1 -. (71) 

The sum over 2 may be upper bounded by an integral, which can 
be evaluated and bounded by simple expressions 






A' 



s — h 



1 < h < s - 1 



A h - in (y/A) s-KH 



2/ 



h — s 



s < h ^ s + 1 
ft > s + 1 



• (72) 



These bounds will be used to bound the right-hand side of (71). 
The first sum in (71) is bounded by the number of terms times the 
largest (last) term. 



2 X exp (X2) < A exp (XA) . 



(73) 



Therefore, defining h„ to be that integer for which h + 1 > s ^ h„ , 
we have 



X h A f 



\ h °A' 



+ < 1 + XA exp (XA) + E (fc . 1)l( ._ fc) + gff^A (fr/4 

(xy)*fe 



■■!■ ■* k ^P L + uw £ ,,„ 

A = A + 2 /i! (« — S) 



A J 



(74) 



In the final sum in (74), /i ^ /i„ -f- 2 > s + 1, and hence the sum may 
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be upper bounded by bounding h/(h — s) by h„ + 2 and then extending 
the summation down to h = 0. Thus, 



= (h + 2)(A/y)'exp{\y). (75) 

Furthermore in the first sum in (74), h ^ h — l^s — 1, and hence 
the sum may be bounded by bounding 1/(8 — h) by 1 and then ex- 
tending the summation to infinity. Thus 

t (k-ml-k) <^t Q $'* A exp (x ^ (76) 

Since s ^ h , 

$ < 1 + 2XA exp (\A) + (s + 2)(A/y)' exp (\y) 

+£^v& WA > + *sf»- C77) 

We shall now choose a particular value for A: 

>-*•-£*(&)■ (78) 

We also assume that L is small enough relative to y so that 

X.A[fc (y/A)] < ^ (79) 

This assumption is equivalent to (57). This condition also ensures 
that X A < 1/eL < 1. The terms of (77) may now be bounded 
separately to yield a convenient upper bound on ^. Thus, 

2\oA exp (M) < 2/L. (80) 

From (78), 

(a + 2)(ii/y)' exp (A„y) - (fi + 2)/L. (81) 

Finally, using (78) and (79) it is easy to show that 



(82) 



^M,/A) + ^X ,<2/L. 

The function tp is now upper bounded for A = A„ by using (80), 
(81), and (82) in (77), 

$ < 1 + 4/L + (s + 2)L = 1 + (6 + s)/L. (83) 
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Then, 

V- - [2 /. exp (\*)] L < [1 + (6 + s)/L] L . (84) 

Now for any L, a ^ 0, 

[1 + a/Lf = 1 + a + UL ~ l) {a/LY 

+ L(L = |p " 2) (a/L) 3 • • • + {a/L)\ 

< 1 4- a + a 2 /2! + • • • = exp (a). (85) 

Hence, 

+ L < exp (6 + s). (86) 

Substituting (86) in (65) , we obtain 

p 2 < HAe/y)' exp (6). (87) 

Finally, after substitution of (87) in (58) and (60) , 



pr 
where 



[EC, ^t/] <DL(Ae/yY, (88) 



D = 1 + e 6 . 



This bound is valid under the condition (57), 

(A/y) (n (y'/A'L) In (y/A) < 1/eL. (57) 
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